Quantity and quality of diagrams used in math word problem solving: A comparison 
between New Zealand and Japanese students 1 



Emmanuel Manalo 

The University of Auckland, New Zealand 
and 

Yuri Uesaka 

The University of Tokyo, Japan 



Paper presented at the NZARE (New Zealand Association for Research in Education) 
National Conference 2006, Rotorua, New Zealand, 5-8 December 



Address for correspondence: Associate Professor Emmanuel Manalo, The Student 
Learning Centre, The University of Auckland, Private Bag 92019, Auckland, New Zealand. 
Telephone: +64 9 3737599 ext 87896. Facsimile: +64 9 3737076. E-mail: 
e . manalo@auckland . ac . nz 



1 Manalo, E., & Uesaka, Y. (2006). Quantity and quality of diagrams used in math word 
problem solving: A comparison between New Zealand and Japanese students. Refereed 
papers of the NZARE (New Zealand Association for Research in Education) National 
Conference 2006 (CD-ROM; ISSN 1 176-4902). Wellington, New Zealand: NZARE. 



1 




Quantity and quality of diagrams used in math word problem solving: A comparison 
between New Zealand and Japanese students 



Abstract 

It is generally considered that diagram use aids efficacy of math word problem solving. 
While understanding diagrams is considered important in both New Zealand and Japanese 
secondary schools, there is an additional emphasis in New Zealand schools for students to 
appreciate their use as tools for problem solving and communication. This study examined 
whether there are actual differences in the amount and quality of diagrams that students in 
New Zealand and Japan use when given math word problems to solve. The participants were 
614 secondary school students from New Zealand and Japan, aged 13 to 15 years old, who 
were given one- and two-object math word problems (involving length or non-length 
components) to solve. The findings were that while the New Zealand students evidenced 
greater diagram use and provided more correct answers, they produced significantly more 
high quality diagrams only for the easiest problems given (the one-object problems with 
length story context). One implication of the findings is that greater emphasis needs to be 
placed on, and/or more effective strategies need to be used in, developing New Zealand 
students’ skills in the use of diagrams for solving more complex types of problems. 



Introduction 

Using diagrams is generally considered by teachers and researchers as one of the most 
effective ways of improving efficacy in solving math word problems. A meta-analysis carried 
out by Hembree (1992) showed that using diagrams was the most efficient of the different 
strategies that had been suggested as helpful for problem solving. Numerous studies (e.g., 
Cheng, 2004; Mayer, 2003; Novick & Hurley, 2001) have reported empirical evidence 
indicating the beneficial effects of diagrams or visual representations on problem solving. 

The usefulness of diagrams can probably be best understood in terms of Larkin and Simon’s 
(1987) explanation that diagrammatic representations are computationally more efficient than 
sentential representations because they minimise labels, gather related information in one 
place, and promote easier recognition of the overall situation being dealt with. Thus, 
diagrams help reduce memory, computational, and searching loads in problem solving. 

Internationally the development of skills relating to diagram use is usually included in the 
math curricula. In Japan, for example, the national curriculum emphasises the crucial nature 
of understanding diagrams and targeted mathematical concepts (Japanese Ministry of 
Education, 1998). Diagrams are likewise emphasised in the New Zealand (NZ) math 
curriculum which has, as one of its objectives, the provision of opportunities for students to 
“develop the ability to think abstractly and to use symbols, notations, and graphs and 
diagrams to represent and communicate mathematical relationships, concepts, and 
generalisations” (NZ Ministry of Education, 1992, p. 10). 

There are some important differences, however, between the Japanese and NZ math curricula 
in the way they view diagrams. One of the most important of these differences is that the NZ 
curriculum goes beyond simply teaching students how to understand diagrams (which is the 
main emphasis in the Japanese curriculum). As noted above, the NZ math curriculum also 
stresses the use of diagrams as a communication tool. It states that: 
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Learning to communicate about and through mathematics is part of learning to 
become a mathematical problem solver and learning to think mathematically. Critical 
reflection may be developed by encouraging students to share ideas, to use their own 
words to explain their ideas, and to record their thinking in a variety of ways, for 
example, through words, symbols, diagrams and models. (NZ Ministry of Education, 
1992, p. 11) 

To this end, the NZ curriculum points out that teachers can create opportunities for students 
to develop the characteristics of good problem-solving techniques (which include both 
convergent and divergent approaches) “by encouraging [students] to practise and learn” 
various simple strategies which include “drawing a diagram” (NZ Ministry of Education, 
1992, p. 11). 

Notwithstanding the apparent importance of skills relating to diagram use in the school 
setting, various studies have described serious deficiencies in students’ abilities to use 
diagrams when problem solving, suggesting that students generally do not appreciate the 
value of diagram use and/or cannot effectively use them. Observations of students’ actual 
work in problem solving have shown, for example, that students lack spontaneity in using 
diagrams (e.g., Dufour- Janvier, Bednarz, & Belanger, 1987; Ichikawa, 1993, 2000; Uesaka, 
2003), that they are generally poor in choosing diagrams to use, and that they fail to draw 
appropriate inferences when they use diagrams (e.g., Cox, 1996; Uesaka, 2003). 

An important question that this raises is: to what extent might the math curriculum - and 
hence the ensuing learning views imparted and teaching approaches utilised - impact on 
student efficacy in using diagrams for problem solving? Both Japan and NZ were ranked very 
high in PISA (Program for International Student Assessment, a project commissioned by the 
Organization for Economic Co-operation and Development [OECD] which assesses the 
knowledge and skills of 15-year-old students from the participating industrialised countries) 
2000 and PISA 2003 (OECD, 2004). However, based on the two countries’ respective math 
curricula, one could predict that NZ students would demonstrate greater spontaneous use of 
diagrams when attempting to solve math word problems. 

To the present authors’ knowledge, no previous investigations have been carried out to 
confinn or refute such a prediction. Thus, the purpose of the research study reported here was 
to examine the spontaneous use of diagrams by students in NZ and Japan when attempting to 
solve math word problems. Apart from the frequency of spontaneous diagram use, the 
correctness of answers provided, as well as the quality of diagrams constructed by the 
students were examined. It was hoped that insights gained through the examination of 
similarities and differences in perfonnance would contribute to the development and further 
refinement of effective teaching practices in both countries. 

Method 

Participants 

Six hundred and fourteen (614) secondary school students from New Zealand and Japan, 
aged 13 to 15 years old (mean ages: New Zealand = 13.97 years, Japan = 13.28 years), took 
part in this study. The New Zealand students (n = 323; female =134, male = 189) came from 
five secondary schools in the upper half of the North Island of New Zealand, while the 
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Japanese students (n = 291; female = 131, male =160) came from four junior high schools in 
the Kanto area in Japan. 

In both New Zealand and Japan, data were collected from schools representing the full range 
of the “ability” spectrum, from what are considered “high level” schools through to those that 
are considered “lower level” schools. This was done to ensure that the range in students’ 
abilities in mathematics was represented in the student groups included in the study. In New 
Zealand, both a top decile school and a bottom decile school (cf. New Zealand Ministry of 
Education, 2005) were included, as well as three schools in between. The math teachers in 
the schools that participated were also requested to administer the problems to approximately 
equal numbers of “high,” “average,” and “low” ability math classes. 

Procedure 

This study was part of a larger study that also examined students’ views about the use of 
diagrams in problem solving, and their responses to suggestions to use diagrams when 
attempting to solve problems they were given. Only procedural details and the students’ 
performance in the first part of the study, which looked at their spontaneous use of diagrams, 
will be reported here. 

A booklet containing math word problems was given to the participants during their regular 
math class. In New Zealand, the booklet was written in English, while in Japan it was written 
in Japanese (various measures were taken to ensure equivalence of the versions). The 
participants were told that the purpose of the research was to find out how they solved math 
word problems, and they were requested to show their working throughout. 

The booklet contained two problems (Problem 1 and Problem 2), one problem to a page with 
ample space below each problem for the participants to work out the solution and provide 
their answer. The participants were told that they had 4 minutes to work on each math 
problem, and not to go to the next problem/page until instructed to do so. There were two 
versions of the booklet which differed only in the story context attached to the math word 
problems (see explanation under Materials below) and, in each class, approximately half of 
the participants were randomly given one version of the booklet while the other half were 
given the other version. One version of the booklet contained math word problems involving 
length, and the other version contained non-length problems. In New Zealand, 173 students 
received the booklet with length problems, and 150 received the booklets with non-length 
problems. The corresponding numbers that received each kind of booklet in Japan were 149 
and 142 students respectively. 

Materials 

Different kinds of math problems were used to ensure they represented the variety that 
students have to deal with in real life. Apart from the length and non-length distinctions noted 
above, in each version of the booklet, two categories of problems were used: one-object 
problems and two-object problems (similar to ones used in Uesaka, 2003). Problems 1 and 2 
in the two versions of the booklet are shown in Table 1. 
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Table 1. The Kinds and Categories of Math Word Problems Used 



Kind 


Category 

One-Object Two-Object 


Length 


You light a candle and it starts to burn. It 
bums at a constant rate, which means that 
it burns at the same speed throughout. 
After 5 minutes, the candle is 10 
centimetres in length. After 7 minutes, it is 
6 centimetres. 

How long does it take for the candle to 
bum out? 


Tom’s house and Hannah’s house are 
connected by one road which is 600 metres 
long. Tom and Hannah talked on the 
telephone and decided to play with each 
other. They leave their own house at the 
same time and start to walk towards the 
other’s house. 

They meet on the road 5 minutes later. The 
place where they meet is 100 metres closer 
to Hannah’s house than the half-way point. 
How fast did Tom walk (per minute)? 

How fast did Hannah walk (per minute)? 


Non- 

Length 


A mouse starts to eat a piece of cheese. 

The mouse eats at a constant rate, which 
means it eats at the same speed throughout. 
After 5 minutes, there are 10 grammes left 
of the cheese. After 7 minutes, there are 6 
grammes left. 

How long does it take the mouse to finish 
eating the piece of cheese? 


There is a pond that can contain up to 600 
litres of water. Now the pond is empty so 
using two taps, A and B, you start to fill 
the pond with water. Water comes out of 
both taps at a constant rate. 

5 minutes later, the pond is full. From tap 
A came 100 litres more water than half of 
all the water in the pond. 

How fast did water come out of tap A (per 
minute)? 

How fast did water come out of tap B (per 
minute)? 



Scoring 

The participants’ work in solving the math word problems was scored by the second author: 
she marked the answers they provided as either correct (P+) or incorrect (P-). She also 
assessed the students’ use of diagrams for each of the problems, and determined in each case 
whether at least one diagram was used (D+) or no diagram was used at all (D-). For this 
assessment, a diagram was defined as any representation of the problem other than words (on 
their own), sentences, or numerical formulas. Tables were counted as diagrams and, for the 
purposes of this study, a table was defined as a depiction of at least a pair of values arrayed to 
represent two related variables. Drawings or illustrations that were considered to be unrelated 
to the problem were categorised as D-. A colleague of the second author’s took the role of 
independent assessor of diagram use in the participants’ work. The inter-rater agreement was 
found to be 96.3%, which the present authors deemed as satisfactory. 

The qualities of diagrams constructed by students were also analysed. Two kinds of criteria 
were used to categorize the diagrams: their structure and the infonnation contained in them 
(see Table 2). If a diagram represented the situation correctly, then it was placed in the higher 
category (category A) for the first criterion. Those that did not represent the situation 
correctly were placed in the lower category (category B). The diagrams were then evaluated 
as to how much relevant infonnation they contained for the second criterion. For example, if 
a diagram contained additional inferences from the word problem then it was placed in the 
highest category (category A); if a diagram contained all of the numbers specified in the 
problem but no additional inferences were evident then it was placed in the second category 
(category B); and, going to the low end of the scale, if a diagram contained no relevant 
numbers at all then it was placed in the lowest category (category E). 
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The diagrams were divided into high and low quality using a combination of these two 
criteria. In other words, diagrams that represented the situation correctly and contained more 
relevant information were categorized as “high quality”, while the rest were categorized as 
“low quality”. To be more exact, for the diagrams produced for the one-object problems, a 
diagram was classified as high quality when it had been placed in category A for both 
structure and content. For the two-object problems, a diagram was classified as high quality 
when it had been placed in category A for structure and in either category A or category B for 
content. The decision about where to place the cut-off point for content was made partly in 
view of the number of diagrams falling into each of the high and low quality categories (i.e., 
a considerably greater proportion of diagrams produced for the one-object problems 
contained appropriate inferences compared to those produced for the two-object problems - 
hence only those that were categorized as A for content were placed in the high quality 
category where the one-object problems were concerned). 

Table 2. The Criteria Used for Evaluating the Quality of Diagrams Students Produced 



for the One- and Two-C 


>bject Problems 


Points of Evaluation 


Categories 


Criteria for Placement in the Categories 


Structure of Diagram 


A 


Represents the situation correctly 


B 


Does not represent the situation correctly 


Information Contained 
in Diagram 


A 


Contains additional inferences drawn from the 
problem 


B 


Contains all of the numbers specified in the 
problem 


C 


Contains some of the numbers specified in the 
problem 


D 


Contains some numbers, but all of them are 
incorrect or unrelated to the problem 


E 


Contains no numbers 



Results 



The percentages of correct answers provided by the NZ and Japanese students to the math 
problems given are shown in Table 3. In all four kinds and categories of problems given, the 
percentages of correct responses provided by the NZ students were significantly higher than 
the corresponding percentages from the Japanese students. 



Table 3. Percentages of Correct Answers from the New Zealand and Japanese Students 



Kind of 
Problem 


Category of 
Problem 


Percentages of Correct Answers 


X(\) value 


New Zealand 
Students 


Japanese 

Students 


Length 


One-Object 


78.03 


49.66 


27.29*** 


Two-Object 


64.74 


49.66 


7.46** 


Non-Length 


One-Object 


70.00 


50.70 


11.38*** 


Two-Object 


65.33 


40.85 


17.58*** 



* p < . 05. ** p < . 01. *** p < .001. 
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The percentages of NZ and Japanese students who spontaneously used diagrams in 
attempting to solve each of the four kinds and categories of problems given are shown in 
Table 4. A significantly higher percentage of the NZ students spontaneously used diagrams in 
attempting to solve one-object problems of the length and non-length kinds. However, this 
was not the case as far as the more complex two-object problems were concerned: no 
significant difference was found between the student groups where the two-object, length 
problem was concerned, and the Japanese students spontaneously used more diagrams in 
attempting to solve the two-object, non-length problem. 

Table 4. Percentages of New Zealand and Japanese Students Who Spontaneously Used 



Diagrams in Af 


tempting to Solve the Problems Given 


Kind of 
Problem 


Category of 
Problem 


Percentages of Students 


X(\) value 


New Zealand 
Students 


Japanese 

Students 


Length 


One-Object 


60.69 


37.58 


17 10*** 


Two-Object 


58.96 


57.05 


0.12 


Non-Length 


One-Object 


48.00 


31.43 


8.28** 


Two-Object 


16.00 


26.06 


4.46* 



* p < .05. ** p < .01. *** p < .001. 



These latter findings were somewhat unexpected and prompted the authors to more carefully 
examine the data that had been collected. This examination revealed that, despite there being 
no difference between the percentages of NZ and Japanese students who produced diagrams 
for the two-object, length problem, the percentage of NZ students (38.67%) who used a 
diagram and got the answer correct for this problem was significantly higher compared to the 
corresponding Japanese students percentage (21.13%), x p) = 10.66, p < .01. Where the two- 
object, non-length problem was concerned, more careful examination of the data revealed 
that the percentage of NZ students (53.33%) who did not use a diagram but produced the 
correct answer was significantly higher than the corresponding percentage of Japanese 
students (33.10%), 'x (i)= 12.15,/? < .001. There was also a significantly higher percentage of 
Japanese students (18.31%) who used a diagram but produced the wrong answer for this 
problem (two-object, non-length) compared to the NZ students (4.00%), x (i) = 15.31 ,P< 
. 001 . 

Table 5 shows the results of the comparison made of the proportion of high quality diagrams 
produced by the students from the two countries. As shown, except where the one-object 
problems with length story context were concerned, the proportion of high quality diagrams 
produced by students from the two countries can be considered equivalent - as no other 
significant differences were found. In one-object problems with length story context, more 
high quality diagrams were produced by the New Zealand students compared to their 
Japanese counterparts. This result suggests that New Zealand students can draw higher 
quality diagrams for the problems which can be considered as easiest to solve. However the 
quality of diagrams produced by students from the two countries did not differ for the more 
difficult problems. 
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Table 5. Percentages of High Quality Diagrams Produced by the New Zealand and 
Japanese Students 



Kind of Problem 


Category of 
Problem 


Percentages of High Quality Diagrams 


X 2 (i) value 


New Zealand 
Students 


Japanese Students 


Length 


One-Object 


80.95 


60.71 


7.76** 


Two-Object 


75.49 


69.41 


0.86 


Non-length 


One-Object 


80.56 


65.91 


3.12 


Two-Object 


41.67 


29.73 


0.92 



* p < .05. ** p < .01. 



To check the validity of the classifications made, the relationship between category 
placement and correct solutions to the problems was analysed. Chi-test analyses revealed that 
the diagrams categorized as “high quality” significantly produced more correct answers for 
the problems given compared to those categorized as “low quality” in both one-object 
problems (length version: X(i) = 41.80 ,p < .01, non-length version: x 2 (i) = 6.28 , p < .01) and 
two-object problems (length version: X (i) = 25.43, p < .01, non-length version: x*(i) = 14.33, 
p < .01). These analysis results confirm the validity of the classification method employed in 
the present study. 

Discussion 

The findings of this study indicate that the NZ students, compared to their Japanese 
counterparts, were better at solving math word problems: significantly more of the NZ 
students provided the correct answers to the four problems given. However, the NZ students 
only spontaneously used more diagrams in attempting to solve the two (length and non- 
length) one-object problems, and the percentage of high quality diagrams they produced was 
only significantly higher where the easiest one-object, length problem was concerned. This 
raises some important questions about what the two groups of students could and could not 
do well as far as math word problem solving and diagram construction were concerned. 

It appears that the NZ students were better at constructing diagrams for the easier math word 
problems: they not only evidenced greater use of diagrams in attempting to solve the one- 
object problems (both length and non-length kinds), they also evidenced a higher percentage 
of high quality diagrams constructed for the one-object, length problem. Their skills in using 
diagrams to solve the more complex two-object problems were more limited - and perhaps 
this is to be expected (since, after all, these problems were more complicated and difficult to 
solve). However, this points to a clear need here for the NZ students to develop skills in 
constructing and using diagrams to help solve more complex math word problems. This same 
need for skills development applies to the Japanese students where both simpler and more 
complex math word problems are concerned. To this end, perhaps the use of strategies such 
as active comparison and review of lessons learnt from problem solving - which Uesaka and 
Manalo (2006) reported as being effective in promoting competence in the construction of 
appropriate diagrams to use - could be trialled and evaluated. 

When dealing with the more complex two-object, non-length problems, a higher proportion 
of the Japanese students used diagrams but produced the wrong answers (suggesting that the 
diagrams they constructed were inappropriate ones). Thus, although there are clear indicators 
that some of the Japanese students were well able to construct appropriate high quality 
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diagrams to use in problem solving, skills development in appropriate choice of diagrams to 
use (an area that both Cox, 1996, and Uesaka, 2003, have identified as being problematic for 
many students) would likely prove beneficial for many. 

One point of interest is that more than half of the NZ students were able to correctly solve the 
two-object, non-length problem correctly without the use of diagrams, as noted earlier. This 
suggests a number of things. First, it confirms that construction of diagrams for solving more 
complex problems is more difficult and requires more attention from teachers if students are 
to develop the knowledge and skills required. Second, it suggests that at least some of the NZ 
students were able to effectively mentally visualise the situation depicted in the more 
complex math word problem, which helped them solve it. Third, it also suggests that at least 
some of the NZ students were able to employ alternative means of solving the more complex 
math word problem (e.g., through the use of algebraic equations) when they were unable to 
construct the appropriate diagrammatic representation. 

The final point above raises the possible question of whether diagrams are even necessary in 
solving more complex math word problems. The answer to this question becomes somewhat 
obvious when one examines the students’ problem solving perfonnance: there is a clear 
decline in production of correct answers for the more complex two-object, non-length 
problem (which coincides with a decline in the students’ use of diagrams). It is of course only 
possible to speculate that had more students been able to construct the appropriate diagram to 
solve the problem in question, more of them would have been able to correctly solve it - but 
such a speculation would be in line with previous findings noted earlier about the benefits 
and/or advantages of using diagrams when problem solving (e.g., Cheng, 2004; Hembree, 
1992; Mayer, 2003; Novick & Hurley, 2001). 

With the performance of the NZ students (compared to the Japanese students) being better on 
various aspects of problem solving and diagram construction, the findings of this study 
suggests that the NZ math curriculum’s greater emphasis on the problem solving and 
communications applications of diagrams does promote the development of the desired skills 
in students. Including the same or similar components in the Japanese math curriculum could 
therefore be beneficial to developing their students’ problem solving capabilities. However, 
as noted earlier, there are some limitations - for example, with the NZ students’ advantage in 
diagram construction and use being limited to the easier problems. This also is only an initial 
investigation (on one age group of students from only two countries), and further studies are 
clearly required to more fully understand the kinds of instruction, learning conditions, and 
motivations that impact on the development of students’ problem solving skills. 
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